<!doctype html public "-//w3c//dtd html 4.0 transitional//en">
<html>
<head>
   <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
   <meta name="GENERATOR" content="Mozilla/4.75 [en] (X11; U; Linux 2.2.14 i686) [Netscape]">
</head>
<body bgcolor="#FFFFFF">

<h1>
The Filter Package: SSF.OS.NetFlow.Filter</h1>

<h3>
Contents</h3>
<A HREF="#Overview">Overview</A>
<br><A HREF="#Implementation"><i>SSF.OS.NetFlow.Filter</i> Implementation</A>
<br><A HREF="#Config">Config Filters in DML</A>
<br><A HREF="#Examples">Examples and tests</A>
<br><A HREF="#Download">Download</A>
<hr WIDTH="100%">
<h3>
<A NAME="Overview"></A>Overview</h3>
The <i>SSF.OS.NetFlow.Filter</i> package is a new package under development,
and it's expected to still have a lot of future changes and updates. The
idea is straight forward: After the NetFlow data is collected and saved,
the nature following step is to try to make use of it. In order to do it
efficiently, it would be nice if the users are provided the way to extract
just those NetFlow records that they are interested in for some more specific
problem, thus here comes the Filter package. This is the package that is
aimed to help the user specify the filter of their own interest and filter
the NetFlow data. The same as the other part of SSFNet, the user can still
config their filters using DML, as shown later below. There are reasons
why the traditional relationship database is not used here. For the NetFlow
data and the application using it, sequential operations counts for an
absolutely majority part. In fact, random accesses are rarely used. Most
of the relationship databases are optimized to have better performance
on random accesses and provide a lot of other things that might not be
very helpful here. Although there are also publications about <i>stream
database</i>, not a very mature product has got into our eyesight yet.
<h3>
<A NAME="Implementation"></A>SSF.OS.NetFlow.Filter Implementation</h3>
The few following lines should seem to be familiar for people who have
the knowledge of BNF or Grammar or compilers.
<br>&lt;filter> = &lt;term> <i>OR</i> &lt;filter> | EMPTY
<br>&lt;term> = &lt;factor> <i>AND</i> &lt;term> | &lt;factor>
<br>&lt;factor> = (filter) | BASECASE | <i>NOT</i> BASECASE
<p>The above few lines are not exactly how the Filter is implemented, but
they are the concepts behind the implementation. As you can see, generally,
a filteris composed of several terms that are connected by logical <i>OR</i>
operation; each term is composed by a few factors connected by logical
<i>AND</i>
operation; and a factor can have <i>NOT</i> operation within it as well
as using another filter. With this general structure, the limitation of
a filter implemented totally depends on the <i>BASECASE</i>, which will
be explained in more detail later below. Together with the <i>inheritance</i>
and <i>replacement</i> features provided by DML, the user can write very
complicated filters of his own.
<p>As for the basecase, let's first consider what a simplest filter will
do. It is given a record, check whether it's within some range, and return
a yes/no answer. The capability of the basecase totally lies in how the
<i>range</i>
can be described and how the <i>check</i> process is done. In the current
package, there is general interface specified for <i>range</i>.
<center><table BORDER WIDTH="100%" BGCOLOR="#CCCCCC" >
<tr>
<td>public interface Range
<br>{
<br>&nbsp;&nbsp;&nbsp; public void config(Configuration cfg, byte dataType)&nbsp;
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
throws configException;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
// describle the range
<br>&nbsp;&nbsp;&nbsp; public boolean inRange(Object data, byte dataType);&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
// check whether the given data is in range.
<br>}</td>
</tr>
</table></center>
There are also two basic ranges provided in the package now. (A third one
is coming out very soon, it should be in the release of SSF.OS.NetFlow.Filter
package as well.) One of them is <i>NumericRange</i>, which is specified
by a minimum and a maximum numerical value. The other is <i>RERange</i>
(Regular Expression Range,it uses the <A HREF="http://www.cacas.org/java/gnu/regexp/">gnu.regexp</A> package by Wesley Warden Bigg.), which is specified by a regular expression.
The third one I mentioned is <i>SetRange</i>, which is speicified by a
given set. Of course the user defined range is also supported now, so that
the user can write their own <i>range</i> class and plug it it.
<p>The basic process of the filter is to extract the data of the relevant
field and check whether it satisfies the requirement. The data extraction
is accomplished with the help of <i>Decoders</i>. Each record type has
its own decoder. Given a field name, the decoder will return the index
of the field data in the byte array received by the filter. It has the
following abstract base class:
<center><table BORDER WIDTH="100%" BGCOLOR="#CCCCCC" >
<tr>
<td>public abstract class Decoder
<br>{
<br>&nbsp;&nbsp;&nbsp; public final static int FIELD_NOT_FOUND = -1;
<p>&nbsp;&nbsp;&nbsp; /**
<br>&nbsp;&nbsp;&nbsp;&nbsp; * If the field doesn't exsit, return FIELD_NOT_FOUND
<br>&nbsp;&nbsp;&nbsp;&nbsp; * the fieldType is also passed in so that
the decoder can check
<br>&nbsp;&nbsp;&nbsp;&nbsp; * whether the fieldType matches the one that
is retrieved by fieldName.
<br>&nbsp;&nbsp;&nbsp;&nbsp; * @ret the index of the field, if not exist,
return FIELD_NOT_FOUND
<br>&nbsp;&nbsp;&nbsp;&nbsp; */
<br>&nbsp;&nbsp;&nbsp; public abstract int getFieldIndex(String fieldName,
byte fieldType);
<br>}</td>
</tr>
</table></center>
To make a demonstration of how the Filter class can be used, there is also
a filterPlayer class within the package. (Actually, the Filter needs to
work with an BasicPlayer object or its decendants.) It's an extended class
of <i>SSF.Util.Streams.BasicPlayer</i>. The user can also write their own
filterPlayer by extending this class. As shown by its name, this player
would "show" those data records that can pass the given filter, and the
filter is configured using a DML file. The following section is the detail
on how to config the filter with DML.
<h3>
<A NAME="Config"></A>Config Filters in DML</h3>
The configuration of a Filter has two main parts. The first part is composed
by the decoders that will be used by this filter, and the other is the
logic part. Below is a detailed explanation.
<center><table BORDER WIDTH="100%" BGCOLOR="#CCCCCC" >
<tr>
<td>filter [
<br>&nbsp;&nbsp;&nbsp;&nbsp; decoder [&nbsp;
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; name ......&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
#the record type name
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; use ......&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
#the decoder class that should be used
<br>&nbsp;&nbsp;&nbsp;&nbsp; ]
<br>&nbsp;&nbsp;&nbsp;&nbsp; ... ...
<br>&nbsp;&nbsp;&nbsp;&nbsp; #------------------------------Other decoders
<br>&nbsp;&nbsp;&nbsp;&nbsp; ... ...
<br>&nbsp;&nbsp;&nbsp;&nbsp; #------------------------------Logic part
<br>&nbsp;&nbsp;&nbsp;&nbsp; term&nbsp; [
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; factor [
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
action ......&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
#deny or permit, the DEFAULT is permit. It's logically the "NOT" operation.
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
field_name ......&nbsp;&nbsp;&nbsp;&nbsp; #the field from where the data
should be extracted
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
field_type&nbsp;&nbsp; ......&nbsp;&nbsp;&nbsp;&nbsp; #the data type of
this field
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
range [
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
#range parameters
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
#if it is Numeric Range
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
min ......
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
max ......
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
#if it is Regular Expression Range, use attribute "reg_exp"
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
#if it is user defined range, use attribute "use" to specify the class
that should be used.
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
#example: use SSF.OS.NetFlow.Filter.SetRange
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;
]
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ]
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; # other factors of
this term
<br>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; ......
<br>&nbsp;&nbsp;&nbsp;&nbsp; #other terms of this filter
<br>&nbsp;&nbsp;&nbsp;&nbsp; ......
<br>]</td>
</tr>
</table></center>

<h3>
<A NAME="Examples"></A>Examples and tests</h3>
<A HREF="../../doc/examples.html#filterTest">Examples and tests documents</A> are here with the examples of the <A HREF="../../doc/monitor.html">SSF.OS.NetFlow</A>
packages.
<h3>
<A NAME="Download"></A>Download</h3>
This <A HREF="http://www.cs.dartmouth.edu/~yuanyg/istswork/download/NetFlow_1.2.5.tar.gz">SSF.OS.NetFlow</A> package has included the Filter package and it should be extracted under $SSF_HOME/src/SSF/OS/. The FilterPlayer is not directly extended from SSF.Util.Streams.BasicPlayer, but it's mostly copied from it, and it will be integrated in the future.
<p>Developed and maintained by Yougu Yuan (<A HREF="mailto:yuanyg@cs.dartmouth.edu"><I>yuanyg@cs.dartmouth.edu</I></A>).
</body>
</html>
